THE SENIOR URBAN EDUCATION RESEARCH FELLOWSHIP SERIES 



Supporting Instruction and Learning 
through Technology in San Francisco 











The Council of the Great City Schools thanks the Institute 
of Education Sciences (IES) for supporting the Senior 
Urban Education Research Fellowship Program. 

The findings and conclusions presented herein are those 
of the authors and do not necessarily represent the views 
of the Council of the Great City Schools or IES. 




The Council of the Great City Schools 






The Senior Urban Education Research 
Fellowship Series 

Volume IV: 

WordSift: Supporting Instruction and Learning 
through Technology in San Francisco 



Kenji Hakuta 
Spring 201 1 



The Council of the Great City Schools is the only national organization exclusively representing the needs 
of urban public schools. Founded in 1956 and incorporated in 1961, the Council is located in Washington, 
D.C., where it works to promote urban education through legislation, research, media relations, instruction, 
management, technology, and other special projects. 



The Senior Urban Education Research Fellowship Series, Volume IV - Spring 201 1 





IV 



The Council of the Great City Schools 




TABLE OF CONTENTS 



Overview: The Senior Urban Education Research Fellowship Program 2 

About the Senior Urban Education Research Fellow 4 

About the Research Partnership 5 

Executive Summary 7 

Introduction: Addressing the Need for Academic Language Development 1 0 

Part I: Development of a Teacher-Centric Language Development Tool 1 4 

Part II: Implementation and Patterns of Usage 20 

Part III: Early Efforts to Evaluate the Efficacy of WordSift. 24 

Discussion 28 

References 30 

Acknowledgements 34 



FIGURES 



Figure 1. Screenshot of the Result of WordSift Text Analysis 1 6 

Figure 2. Effect of Web-Based Vocabulary Tool 

on Reading Comprehension Assessment Scores by Gender 26 



The Senior Urban Education Research Fellowship Series, Volume IV - Spring 201 1 







OVERVIEW 



THE SENIOR URBAN EDUCATION 
RESEARCH FELLOWSHIP PROGRAM 

Large urban public school districts play a significant 
role in the American education system. The largest 65 
urban school systems in the country - comprising less 
than one half of one percent of the nearly seventeen 
thousand school districts that exist across the United 
States - educate about 14 percent of the nation's 
K-12 public school students, approximately a quarter 
of its economically disadvantaged students, a third of 
its African American students, a quarter of its Hispanic 
students, and a third of its English Language Learners. 1 
Clearly, any attempt to improve achievement and to 
reduce racial and economic achievement gaps across 
the United States must involve these school districts as a 
major focus of action. 

These school districts face a number of serious, 
systematic challenges. To better understand the problems 
in urban education and to develop more effective and 
sustainable solutions, urban districts need a program 
of rigorous scientific inquiry focusing on what works 
to improve academic outcomes in the urban context. 
Moreover, in order to produce such evidence and to move 
public education forward generally, the standards of 
evidence in education research must be raised in such a 
way as to bring questions regarding the effectiveness of 
educational interventions and strategies to the fore and 
to promote careful scrutiny and rigorous analysis of the 
causal inferences surrounding attempts to answer them. 

It has been argued that, in order to move such an effort 
forward, a community of researchers, committed to a 
set of principles regarding evidentiary standards, must 
be developed and nurtured. We contend further that, in 
order to produce a base of scientific knowledge that is 
both rigorously derived and directly relevant to improving 
achievement in urban school districts, this community of 
inquiry must be expanded to include both scholars and 
practitioners in urban education. 



Though a great deal of education research is produced 
every year, there is a genuine dearth of knowledge 
regarding how to address some of the fundamental 
challenges urban school districts face in educating 
children, working to close achievement gaps, and 
striving to meet the challenges of No Child Left Behind. 
Moreover, while there is a history of process-related 
research around issues affecting urban schools, relatively 
few studies carefully identify key program components, 
document implementation efforts, and carefully examine 
the effects of well-designed interventions in important 
programmatic areas on key student outcomes such as 
academic achievement. In sum, there is an absence of 
methodologically sound, policy-relevant research to help 
guide practice by identifying the conditions, resources, 
and necessary steps for effectively mounting initiatives 
to raise student achievement. 

In order to address this need, the Council of the Great City 
Schools, through a grant from the Institute for Education 
Sciences, established the Senior Urban Education 
Research Fellowship (SUERF) program. 

The Senior Urban Education Research Fellowship was 
designed to facilitate partnerships between scholars and 
practitioners focused on producing research that is both 
rigorous in nature and relevant to the specific challenges 
facing large urban school districts. We believe such 
partnerships have the potential to produce better, more 
practically useful research in at least three ways. First, 
by deepening researchers’ understanding of the contexts 
within which they are working, the program may help them 
maximize the impact of their work in the places where it is 
needed the most. Second, by helping senior staff in urban 
districts become better consumers of research, we hope 
to increase the extent to which the available evidence 
is used to inform policy and practice, and the extent to 
which urban districts continue to invest in research. Third, 
by executing well-designed studies aimed at the key 
challenges identified by the districts themselves, we hope 
to produce reliable evidence and practical guidance that 
can help improve student achievement. 



Council of the Great City Schools (2010). Beating the Odds: An Analysis of Student Performance on State Assessment and NAEP. 
Results from the 2008-2009 School Year. Washington, DC. 
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The primary goals for the Senior Urban Education 
Research Fellowship are to: 

• promote high quality scientific inquiry into the ques- 
tions and challenges facing urban school districts; 

• facilitate and encourage collaboration, communi- 
cation, and ongoing partnerships between senior 
researchers and leaders in urban school districts; 

• demonstrate how collaboration between scholars 
and urban districts can generate reliable results 
and enrich both research and practice; 

• produce a set of high quality studies that yield 
practical guidance for urban school districts; 

• contribute to an ongoing discussion regarding 
research priorities in urban education; and 

• promote the development of a “community of 
inquiry”, including researchers and practitioners 
alike, committed to both a set of norms and prin- 
ciples regarding standards of evidence and a set 
of priorities for relevant, applied research in urban 
education. 

The SUERF program benefitted greatly from the guidance 
and support of a Research Advisory Committee made up 
of experts and leaders from large urban school districts 
and the education research community. The committee 
included Dr. Katherine Blasik, Dr. Carol Johnson, Dr. Kent 
McGuire, Dr. Richard Murnane, Dr. Andrew Porter, and 
Dr. Melissa Roderick. This extraordinary group helped to 
identify and define the objectives and structure of the 
fellowship program, and we thank them for lending their 
considerable insight and expertise to this endeavor. 



The following volume of the Senior Urban Education 
Research Fellowship Series documents the work of 
Dr. Kenji Hakuta, working in collaboration with the San 
Francisco Unified School District under the auspices of 
the Strategic Education Research Partnership (SERP). 
Both the research and reporting is the sole intellectual 
property of Dr. Hakuta, and reflects his personal 
experience and perspective as an education researcher 
working in collaboration with SFUSD. 

Dr. Hakuta’s work developing and piloting the WordSift 
tool in San Francisco makes an important contribution 
to the growing field of instructional technology. In 
recognition of the role academic language plays in 
student achievement, he has designed this tool as a 
resource for content area instructors, helping them to 
become teachers of the academic language of their 
discipline. The need to support and advance academic 
literacy instruction across the curriculum is a fundamental 
challenge facing researchers and educators alike, and 
technology and innovation will likely play a significant role 
in our efforts to reform and improve. We hope you will 
find this report both interesting and relevant to your own 
work in urban education. 

Thank you. 

Michael Casserly 

Executive Director 

Council of the Great City Schools 
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University. An experimental psycholinguist by training, he is best known 
for his work in the areas of bilingualism and the acquisition of English in 
immigrant students. He is the author of numerous research papers and 
books, including Mirror of Language: The Debate on Bilingualism and 
In Other Words: The Science and Psychology of Second Language 
Acquisition. He chaired a National Academy of Sciences report Improving 
Schooling for Language-Minority Children, and co-edited a book on affirmative action in higher education, Compelling 
Interest: Examining the Evidence on Racial Dynamics in Higher Education. Hakuta is also active in education policy. 
He has testified to Congress and other public bodies on a variety of topics, including language policy, the education of 
language minority students, affirmative action in higher education, and improvement of quality in educational research. 
He has served as an expert witness in education litigation involving language minority students. Hakuta received his BA 
Magna Cum Laude in Psychology and Social Relations, and his Ph.D. in Experimental Psychology, both from Harvard 
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THE STRATEGIC EDUCATION 
RESEARCH PARTNERSHIP 

This work was conducted in the context of a larger 
research partnership at the San Francisco Unified 
District organized by the Strategic Education Research 
Partnership (SERP). SERP was conceived and incubated 
at the National Academy of Sciences, and began operating 
as an independent institute in 2005. Its ambitious 
mission is to create an infrastructure for problem-solving 
research, development, and implementation in education. 
The SERP partnership model is unique in several 
respects: 

• The SERP model assumes that the 
development and maintenance of effective 
partnerships between researchers and 
practitioners is itself a form of expertise 
that must be developed and nurtured 
over time. Knowledge and experience must 

be accumulated regarding the trajectory of 
trust building, and the negotiation of agendas. 
Policies and practices must be developed that 
place the district in the lead on defining problems 
for attention, and researchers in the lead on 
designing research. It must nurture a sense of 
shared responsibility for designing approaches 
to improvement. Expertise must be developed on 
the types and frequency of interaction required 
for sustained commitment to the partnership, the 
participants who must be at the table to support 
progress, the sources of competing attention that 
must be taken into account, and the predicable 
obstacles that can derail the effort or diminish its 
productivity. Thus, a major SERP role is forming 
and maintaining researcher-district relations. 

• The SERP model assumes that a third party 
organization charged with coordination 
can ensure the goals of the partnership 

are primary at all times. There are strong 
incentives for both researchers and education 
practitioners to respond to pressures and rewards 
in their own institutional and professional contexts. 
And if university researchers or school district 



administrators fail in the collaborative effort, their 
primary source of professional identity is largely 
unaffected. A coordinating organization fails if the 
collaboration fails, creating incentives to persist 
through the most difficult challenges, and search 
for solutions that will strengthen the long run 
prospects of the collaboration. 

• The SERP Model assumes that building 
a knowledge base will require an active 
effort to link sites-not just a sharing of 
information, but a reshaping and retesting 
of ideas. The coordinating SERP organization 
takes on responsibility for ensuring that each 
new site has access to the research tools and 
instruments developed in other sites, and that 
knowledge and experience that emerge from 
one site, as well as the interventions and tools 
developed, are actively carried to other sites. 

SERP AND THE SAN FRANCISCO 
UNIFIED SCHOOL DISTRICT 

The existence of the SERP-San Francisco field site within 
the San Francisco Unified School District (SFUSD) was 
crucial to the development and initial testing of WordSift. 
The SERP partnership with the district began in January 
2007, and the primary focus of the SERP-SFUSD 
work during the period of this project was defined as 
addressing the achievement gap in middle schools, 
especially in math and science, and the language and 
literacy needed to support the closing of the gap. The 
problem was defined through the District’s recognition 
that its gap between high- and low-achieving students is 
among the highest in California urban districts, and that 
much of the low end of the achievement spectrum was 
due to the performance of African-American and Latino 
students. 

The SERP work in science and science literacy is built 
on the premise that narrowing that gap will require a 
high level of specificity regarding what students who 
do well in science know, and what those who do poorly 
need to know. The work makes a further assumption that 
much of what is taught is not critical to mastery of the 
core ideas, and that progress in narrowing the gap will 
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require the identification of a subset of “core concepts” 
that should be given intensive focus in all classrooms. 
Finally, it is assumed that teachers need a set of tools 
and instructional strategies to teach this core content 
and monitor student learning effectively. 

Over the past two years through these SERP activities, 
the science co-developer teachers, district supervisors 
and administrators, and researchers have moved toward 
a system of knowledge and trust-building that begins 
to address these important issues in instruction and 
assessment. Through work with the Berkeley Evaluation 
& Assessment Research Center (BEAR) at University 
of California at Berkeley, (Mark Wilson, Director), the 
co-developers have created explicit progress maps in a 
number of domains of science instruction in the 6th and 
8th grade that correspond to learning progressions for 
students. 

In this context, a group of us working on the language 
characteristics and demands of the curriculum have 
elevated teacher awareness of these issues and taken 
the first steps in addressing these gaps. Professor 
David Pearson and Post Doctoral fellow Marnie Nair 
from University of California at Berkeley participated 
in the initial work in identifying language needs. Karen 
Thompson, a graduate student from Stanford, provided 
early analysis of the linguistic complexity of the science 
textbooks being used. WordSift was created as part of 
this effort, to help address the recognition that teachers 
of science needed tools help to become teachers of the 
academic language of their discipline. 



The costs of developing and piloting this tool were 
covered by grants to SERP and the present fellowship 
grant from the Council of Great City Schools through 
the Senior Urban Research Fellowship Program, funded 
by IES, from 2007 to 2009. The previously established 
relationship with the SFUSD administration and with the 
instructional leaders in SFUSD schools made possible 
a highly collaborative process for developing WordSift, 
which included tool design and developing uses for 
teachers and students. SERP has supported the use of 
this tool with the current group of teacher co-developers, 
having them use the tool in 6 collaborating schools, 
collecting and processing feedback from teachers in 
those schools, and incorporating the feedback into the 
design of the tool. 

SERP played a key role in the work described here. SERP 
has built a strong framework to facilitate communications 
between the practitioners in SFUSD and the research 
team so that there is a regular feedback loop between 
the district and the researchers. These include period 
meetings focused on the design nature of the research 
and practice work, as well as subject-specific working 
groups that oversee the direction and progress of the 
research work. In addition, a core team of district and 
research leaders meets regularly to discuss the needs of 
district and the progress of ongoing research efforts. The 
core team provides the general guidance and permission 
for work within the school district so that the research 
work is pertinent to the needs of the teachers and 
principals and aligned to the goals of the district strategic 
plan. 
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EXECUTIVE SUMMARY 



In developing a tool to help address academic language 
development, the team began by identifying the challenges 
of presenting science text to English Language Learners 
and students who struggle with reading. The goal was 
to balance the practical and logistical challenges of 
identifying, teaching, and learning academic and content 
vocabulary with the opportunities posed by web-based 
visualization technologies. For example, content area 
teachers do not always feel comfortable teaching 
language and vocabulary, and may require instructional 
prompts and examples of sentences in which specific 
vocabulary appear to scaffold their role as language 
teachers. 

We involved teachers early on in this development 
process, and through a series of co-developer meetings 
with a panel of science teachers in the San Francisco 
Unified School District, endeavored to create a truly 
teacher-centric instructional tool. We also confronted the 
realities of the urban school setting, where resources are 
often a challenge and teachers vary in their experience 
and comfort level with classroom technology. 

The result was WordSift, a web-based interface that 
creates a visual, interactive representation of selected 
text. Based on any English text entered into a “copy and 
paste” box, WordSift rapidly returns a visual display— or 
Tag Cloud— that displays the top 50 words in alphabetical 
order, with the relative frequency of each word indicated 
by text size. The order of words in the cloud can be 
rearranged dynamically in a number of ways, including 
by frequency in the English language, by those most 
commonly appearing in subject matter areas, and by 
those belonging to various academic word listings or 
glossaries. WordSift also produces related Google Image 
and Video Searches, a Visual Thesaurus® display, and 
example sentences. 



Through regular meetings with our panel of teacher 
collaborators, a number of potential uses of WordSift 
were identified as useful to support instruction for 
teachers and reading comprehension for students. These 
included lesson preparation, the development of group 
activities for teachers, and the ability to preview text and 
access additional literacy support for students. WordSift 
was launched to the public on January 15, 2009. Usage 
statistics and teacher reviews of WordSift are discussed 
briefly in Section II. 

In addition to developing WordSift and looking at how 
it could be applied in the classroom, our effort also 
extended to the development of research to evaluate 
the effectiveness of WordSift on student reading 
comprehension of science text. In the spring of 2010, 
we implemented a systematic study with the science co- 
developers to see whether previewing a science text with 
WordSift would improve the reading comprehension of 
their students relative to when they previewed text by other 
means. Early results suggest that the variation observed 
in reading comprehension assessment scores was not 
significantly related to differences based on exposure to 
WordSift, but rather to differences among characteristics 
of the students and the classes and schools in which they 
are enrolled. 

Interestingly, gender appeared to significantly mediate 
the treatment effect, resulting in a significant negative 
treatment effect for girls and a slightly positive (though not 
significant) treatment effect for boys. For future iterations 
of the web-based vocabulary tool, we may choose to pilot 
test it specifically with girls to get their feedback about 
design and features that appeal to them. 
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Although disappointing, it is important to remember that 
this is just one implementation of WordSift (a whole- 
class demonstration, followed by a reading passage and 
overall comprehension of the text), and only a first attempt 
to investigate the tool’s impact on learning. We discuss 
several avenues for future evaluation, including studying 
the use of the tool in classrooms with more English 
learners, looking at long-term vocabulary retention, and 
testing the effect of each of the different functions of 
WordSift individually. 

In conclusion, we try to put our work developing, 
implementing, and evaluating WordSift into context. While 
this tool represents the potential instructional technology 
holds to transform classroom learning and advance 
academic vocabulary development, the two pillars of the 
project - language and technology - are really means to 
the end of access to subject matter content. To address 
this challenge, we need to build a culture in schools and 
districts where all teachers take responsibility for the 
language development of their students, and have access 
to the resources they need to approach this task. Districts 
and researchers alike would also benefit from involving 
teachers as collaborators and key stakeholders in the 
process of identifying new, innovative classroom tools 
for increasing student achievement and engagement in 
learning. 
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INTRODUCTION 



INTRODUCTION 



ADDRESSING THE NEED FOR 
ACADEMIC LANGUAGE DEVELOPMENT 

The power of vocabulary in predicting cognitive command 
of phonological, orthographic, and semantic processing 
as well as reading rates and other tasks are well 
documented (Nation & Snowling, 2004; Yang & Perfetti, 
2006). Walter Kintsch's work on situation modeling from 
text also demonstrates the importance of higher order 
reading comprehension abilities and its relationship 
to vocabulary (Kintsch, 2086; Perfetti, Landi & Oakhill, 
2005), while Catherine Snow and her colleagues 
have demonstrated increasing correlations between 
vocabulary scores and reading comprehension scores as 
student move from primary to secondary grades (Snow, 
Porche, Tabors & Harris, 2007). 

The challenges of vocabulary are now fairly well mapped. 
By the middle school grades, students are expected 
to read and understand expository texts in various 
content areas with demanding vocabulary (Gardner, 
2004). There is often a “Matthew effect,” with the rich 
getting richer with respect to vocabulary and, ultimately, 
reading comprehension (Nagy & Anderson, 1984; Nagy 
& Herman, 1987; McKeown, 1985; Stanovich, 1986; 
Swanborn & de Glopper, 2002). 

Fortunately, there is evidence that vocabulary instruction 
can have an important and lasting impact on student 
word learning (Beck, Perfetti & McKeown, 1 982; Carlo 
et al, 2004). And while this is true for all students, it is 
especially true for English Language Learners. 



Of special interest to researchers and educators seeking 
to address the challenge of academic literacy and 
vocabulary are academic words (Coxhead, 2000) that 
cut across subject matter areas (e.g., affirm, interpret, 
deny, evidence, conclusion, theory, factor, process) in 
contrast to subject-specific words such as mitosis, plate 
tectonics or prepositional phrase. These words are of 
special interest because they are unlikely to be identified 
in glossaries in many subject area textbooks, yet they are 
crucial for understanding the meaning of the text. 

Other vocabulary categories of interest are subject- 
specific lists (e.g., Marzano & Pickering, 2005), such as 
Biemiller's list of “Words Worth Teaching," based on the 
work of Isabel Becks and her colleagues on the Living 
Word Vocabulary and his own empirical work identifying 
key words (Beck, McKeown & Kucan, 2002; Biemiller, 
2005, 2006). With increased capacity for automated 
word frequency counts in an ever-increasing database 
of texts, there is now increased capacity in the field to 
target specific words for specific students, based on their 
backgrounds and the instructional subject area. Such 
lists enable explicit instruction of specific vocabulary in 
the context of content area instruction. 

In addition to the challenges of academic vocabulary 
development, educators specializing in English Language 
Learners are faced wifh an even greater instructional 
challenge: a substantial group of students who are “stuck” 
at the intermediate level of English language proficiency, 
leading to labels such as long-term ELLs or “lifers”. 2 



2 These students most likely face similar challenges as students who are monolingual English-speakers (many speakers of non-Standard English) do, 
but accentuated by their classification as ELLs. 
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From a pedagogical viewpoint, an important key to 
addressing this problem lies in the realization that 
English language development for these students - while 
the primary responsibility of the ESL/ELD teacher or 
specialist, especially for ELL students in the beginning 
levels of English language development - needs to be 
an overarching objective for all teachers of English 
Language Learners - the so-called “mainstream” teachers 
in elementary school, or the subject area teachers in 
language arts, math, science, social studies, and electives 
in the secondary grades. Language development needs 
to occur in context, and content area learning provides the 
best context for it to occur. 

But can we expect mainstream and content area teachers 
to take on this shared responsibility? Teacher practice is 
the product of a complex set of situations, beginning in 
teacher preparation programs (in which considerable 
separation of responsibilities for “mainstream” and “ELL 
students” take root) and the culture of the school created 
by its demographic composition, district expectations and 
the site leadership and professional support. 



A key challenge is to help content teachers define a new 
identity for themselves - as a language teacher of their 
discipline. 

Addressing this challenge of helping regular content 
teachers become effective teachers of academic 
language and vocabulary is the main goal of this project. 
Our work in San Francisco public schools addresses one 
of the greatest challenges facing educators of English 
Language Learners: how to grow and enrich the academic 
vocabulary of their students across the grade levels, and 
especially through academic content instruction. To this 
end, our goal is to provide language tools and strategies 
that can be easily embedded into their existing practice 
as content teachers. 
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PART I: DEVELOPMENT OF A TEACHER-CENTRIC 
LANGUAGE DEVELOPMENT TOOL 



In developing a tool to help address this need for 
academic language development, the team began by 
identifying the challenges of presenting science text to 
English Language Learners and students who struggle 
with reading. In particular, the team focused on the 
practical and logistical tasks of identifying, teaching, and 
learning academic and content vocabulary. 

For example, content area teachers do not always feel 
comfortable teaching language and vocabulary, and 
require instructional prompts and examples of sentences 
in which specific vocabulary appear to scaffold their role 
as language teachers. In addition, teachers rarely have 
time to read through whole science texts in preparation 
for a lesson and pick out the top academic or high 
frequency words, nor do they necessarily know what the 
academic words are in selected texts. 

At the same time, students who identify unfamiliar 
vocabulary often turn to a dictionary or thesaurus for its 
definition. For English learners in particular, one of the 
strategies developed for Specially Designed Academic 
Instruction in English (SDAIE) is supporting vocabulary 
development through associating words with images 
and visualizations. Yet in both the traditional hardcopy 
and online formats, dictionary definitions are static, one- 
dimensional, and include phrases or sentences that 
may or may not support students' understanding of the 
word. Similarly, a thesaurus can confound students with 
static lists of words related to the original word but no 
more meaningful to the student. The dynamic nature of 
words, and the relationships to images, concepts, and 
other words can easily be obscured by these traditional 
formats. 

The team endeavored to balance these types of student 
and teacher challenges with the opportunities posed 
by web-based visualization technologies. The result 
was WordSift, a web-based tool that creates a visual 
representation based on any English text entered into a 
“copy and paste" box, rapidly returning a visual display 
that allows the user to explore the text. There are other 
visual tools that have served as predecessors, including 
TagCrowd (http://tagcrowd.com/), Wordle (http:// 
www.wordle.net/), and VocabGrabber (http://www. 



visualthesaurus.com/vocabgrabber/), but this is the only 
site that integrates web-based images and is developed 
from the very beginning with the objective of supporting 
K-12 teaching and learning. 

Based on the idea and direction of the principal 
investigator, Kenji Hakuta, the website was produced 
by Greg Wientjes, a doctoral student in Stanford's 
Learning Design and Technology program. Two doctoral 
students with significant prior experience in teaching 
English Language Learners - Diego Roman and Karen 
Thompson - provided additional design support to make 
the site friendly to teachers. 

The development process also benefitted greatly from a 
series of co-developer meetings with a panel of science 
teachers in the San Francisco Unified School District 
during the 2008-2009 school year. Initially, the teachers 
were asked for reactions and observations, and through 
a continuous feedback process, a number of potential 
uses of WordSift were identified as useful to support 
instruction for teachers and reading comprehension for 
students. For example, the initial version of WordSift 
only marked words from the Academic Word List, but 
teachers expressed a strong interest in having discipline- 
based words marked as well, a feature that was added. 

We also confronted the realities of the urban school 
setting, where high quality LCD projectors were not 
uniformly available (some teachers, for example, had 
to print out the output onto overhead transparencies in 
order to share the WordSift output) and the bandwidth 
of computer labs was not sufficient for all students to 
simultaneously connect to the Internet. We also aimed to 
take into account the fact that teachers vary considerably 
in their comfort level with technology, which figured 
largely into the design principals we followed when 
creating the tool. Specifically, we aimed to maximize the 
uptake by a broad group of teachers by: 

• Keeping it simple, with minimal clutter on the 
screen and maximum ease of use, requiring only 
the ability to browse, cut, and paste. 

• Maximizing the visual experience by linking 
words in a visually appealing way with images. 
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• Making it fast, as long wait times are frustrating 
for both students and teachers and disrupt the flow 
of instruction. 

• Being interactive, spontaneous and engag- 
ing, encouraging spontaneous talk, interaction and 
queries. 

• Being projector-friendly. WordSift screenshots 
are designed to also yield functional screen projec- 
tions for classrooms using a variety of projectors. 
This function has also served to remind us not to 
add a lot of clutter to the screen, as this would limit 
its horizontal size. 

• Creating a “wow!!!” sensation. This may not 
sound very scientific, but we value this as the initial 
reaction of first-time users of WordSift, which is 
then followed by the interaction with the text that 
allows deeper processing of its meaning. 

A Closer Look at the Design Features 
of WordSift 

The easiest way for fhe reader to understand the features 
of WordSift would be to go online and try it out - http:// 
www.WordSift.com. WordSift takes text that is pasted into 
the input box and creates the following displays: 

Tag Cloud 

A Tag Cloud of the top 50 words in the text, excluding 
function words (such as is, of, at, the - the full list of “stop 
words” that are blocked from the display can be found 
under Word Lists on the home page of WordSift). The 
words in the cloud are initially displayed in alphabetical 
order, and their relative frequency is indicated by text size. 
Singular and plural nouns are, for the most part, counted 
together. The order of words in fhe cloud can be rearranged 
dynamically by frequency in the English language by 
hitting the Common-to-Rare or Rare-to-Common button 
at the bottom of the cloud display. Currently, we use the 
word frequency list from Projecf Gutenberg because of its 
sheer size (it is the word frequency count from all books 
in Google’s ambitious effort to make all books available 
online), although we are considering adding a list more 
targeted to textbooks. 



Words in the cloud can also be marked as those most 
commonly appearing in subject matter areas. We have 
created an initial list from language arts, math, science 
and social studies. Words from the General Service List 
(GSL) and the Academic Word List (AWL) can also 
be marked. This dynamic quality allows the teacher or 
student to quickly see the key vocabulary in the text, look 
for low frequency words that could cause difficulty for fhe 
student, and identify key academic words. 

This tag cloud solves several problems: it saves time for 
teachers in populating the top frequency text; it highlights 
targeted vocabulary types (the academic word list, the 
low frequency words, and subject-specific words) that 
are different from the subject area glossaries typically 
highlighted in textbooks; and it creates a quick way for 
the teacher to assess whether any key vocabulary words 
might present a challenge to their students. 

Google Image and Video Searches 

The results of Google searches are also displayed, 
automatically entering the two most frequent words as 
the search terms. The top 8 results from the images 
search are displayed. The search can be directly changed 
by entering or deleting words in the “Search" box in the 
panel. Also, clicking on any word in the tag cloud will 
automatically refresh the display with the search result for 
that word. 

We have found thaf the image search frequently results 
in photos that are of very close relevance to the topic of 
the text. Even results that seem to be somewhat off base 
can be used as a way of talking about why the program 
might have chosen it. For example, Figure 1 demonstrates 
the results of a search on “Darwin and Evolution," which 
produced a cloud with “Darwin” and “selection” as the 
top two words, as well as pictures representing evolution. 
The teacher can use this to start a discussion about 
images that represent the concept of natural selection. 
Further refinement of the search can be made by typing 
in additional or different key words into the search box - 
changing it to “natural selection” gets more images that 
get closer to representations of the process of natural 
selection. 
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The video results, meanwhile, are “hidden” and can be 
accessed by clicking on “Videos" below. This is because 
the images can be strictly filtered to avoid inappropriate 
content for students, whereas the videos are less 
predictable. Many school districts block YouTube, but this 
can be a very rewarding part of the exercise if the teacher 
can use it as a source of talk about language, rather 
than as a distraction. If teachers preview videos in an 
unblocked environment (such as at home or a different 
server at school) then they can download appropriate 
videos to show in class. We are currently exploring ways in 
which unblocked sites, such as TeacherTube and various 
public broadcasting websites, can be easily accessed on 
WordSift. 



This function of Google Images and Video Searches 
provide additional supports for students so they are not 
confronted with only text definitions of vocabulary, but 
related visual representations of the vocabulary. Also, 
while abstract words may be difficult to display in images 
- words such as “respect” or “imagination," Google Images 
has developed a method through its Google Image 
Labeler that essentially creates a database of images 
that have strong association across a large number of 
individuals. This enables a discussion of abstract words 
using pictures, and students and teachers are able to talk 
about how people would have paired the abstract word 
with the resulting images. 



FIGURE 1 ; SCREENSHOT OF THE RESULT OF WORDSIFT TEXT ANALYSIS 
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Visual Thesaurus® 

he most frequent word is entered into the Visual Thesaurus 
® and the result is displayed as a word web. The Visual 
Thesaurus is a product based on WordNet, a digital 
dictionary and thesaurus created by George Miller and his 
colleagues (Fellbaum, 1 999). In WordSift, the free widget 
made available from the Visual Thesaurus is embedded 
(with permission). The VT widget displays the word, plus 
related words including antonyms and synonyms. 

The Visual Thesaurus® display is interactive: the 
definition of each word on the display pops up when the 
cursor is scrolled over it, and a click on any word on the 
web re-configures the display to bring that word to the 
center. This display is just a sampler taken from the Visual 
Thesaurus® website and does not contain all of the other 
features available in the full product. By clicking on the full 
version, these additional features can be accessed. 

One potentially useful feature enables the user to hear 
the word by clicking on the speaker icon next to the word. 
Another feature will provide an overlay of words from 
selected languages, including Spanish. It is a potentially 
powerful way of working with bilingual students on 
vocabulary development, particularly in identifying 
cognates, and it can be used to enhance development 
in both languages. Use of these full features is limited to 
approximately 7 attempts without purchasing the product. 
After that, going back to WordSift and re-initiating the 
open the full version button will refresh the function. 

The VT creates an interactive platform for students to 
visualize the relationships between words. The sound 
feature on the full version of VT is also helpful in 
helping students hear the pronunciation of words, since 
most students (or adults, for that matter) have difficulty 
decoding the phonetic representation of words in regular 
dictionaries. 



Example Sentences 

Examples from the source text containing the most 
frequent word in the text is displayed under the Visual 
Thesaurus word web. The key word is marked in green. All 
relevant examples from the input text are listed. 

One intended use of this feature is to organize the text 
to preview key vocabulary. This feature can quickly show 
different meanings of the same word so that, when 
presented with the full list of sentences from the text in 
which a given word appears, teachers can talk about the 
different sentence frames and about different nuances 
of the word. For example, in an article on climate change, 
here are two sentences that use “area”: (1) Weather 
scientists project that the polar regions of the Northern 
Flemisphere will heat up more than other areas of the 
planet, and glaciers and sea ice will shrink as a result. (2) 
The area covered by sea ice during summer has declined 
by 1 5 to 20 percent in the last 30 years, and is projected 
to disappear almost completely late in the 21st century. 
The first meaning of area is “region” whereas the second 
meaning refers to a specific quantity, in terms of surface 
area. A look at “area” under in the Visual Thesaurus box 
shows that these are two of the major distinctive branches 
from the word. 

Tailoring the Results 

The Google results, the VT, and the Example Sentences 
are all dynamically linked to the word clicked on the tag 
cloud, enabling rapid exploration of the features of any 
word. We are presently working on a feature that would 
enable “drag-and-drop" capabilities into a workspace, 
so that words and images can be collected, saved, and 
printed. This would enable the teacher to create quick 
assessments, and for students to create quick outlines for 
writing assignments. 
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As mentioned in the previous section, WordSift was 
developed through an extended collaboration with 
teachers and administrators from the San Francisco 
Unified School District (SFUSD) under the auspices of 
the Strategic Education Research Partnership (SERP). 
Our teacher collaborators included four 6th grade 
science teachers (Karen dayman, Lisa Ernst, Patricia 
Kudritzki, and Stephie Prout) and five 8th grade teachers 
(Joel Austin, David Brody, Michael Fox, Sally Meneely, 
and Lisa Beth Watkins) supported by SFUSD Middle 
School Science Curriculum Specialist Deborah Farkas, 
Mathematics and Science Supervisor Jeanne DArcy, and 
SERP/SFUSD Assistant Director Tina Cheuk. 

Teacher reviews of their participation in this process were 
obtained systematically for some sessions. The regular 
collaborative meetings resulted in the identification of 
some important uses for WordSift : 

Lesson preparation: A teacher can use WordSift 
to review assigned text to identify challenging words 
or concepts prior to a lesson, and identify images and 
videos to use in class. The videos can be especially useful 
in the preview function since many schools do not allow 
access to YouTube, but a teacher can download useful 
videos (such as a science iab demonstration) onto his or 
her laptop computer from home. 

Previewing text: In a whole class setting or individually, 
students can preview text. Reading comprehension 
research suggests that previewing text is a useful strategy 
for improving comprehension. Using WordSift to identify 
the key vocabulary and playing with the images and to 
use the example source sentence feature to “skim” the 
text can help students who might otherwise struggle with 
the complexity of the text. 



Group activities: Teachers have found simple activities 
using small portions of WordSift useful. For example, 
one teacher has developed a simple routine in which she 
gives students the TagCloud, and has them working in 
small groups to write or draw a page using the words in 
the cloud. Another possibility would be to take the Visual 
Thesaurus display of a word web and have students 
identify and discuss related words. 

Literacy support: Individual students can use WordSift 
as they read text, or as they write a response or summary. 
Adult users of WordSift have reported using WordSift 
for their own purposes to skim text (as one teacher said, 
“I don't skim, I sift”) and also to review their own writing 
drafts. 

Assessment: Whole-class vocabulary assessment 
can be done on-the-fly by showing the images from 
selected words, having them identify unfamiliar words, 
and having students talk about which picture is the 
best representation of a given word. Teachers can also 
tailor their own assessments by copying and pasting the 
images, words, and sentences identified by WordSift 
into a separate file (such as in Word or Powerpoint) and 
printing it out for student work. 

Teacher evaluations were gathered at the end of each 
workshop to identify lingering questions and concerns. 
For example, it was quite clear that discipline-oriented 
teachers had some difficulty seeing the value of focusing 
on academic words rather than the words of their 
discipline, and this resulted in a clearer explanation of 
this distinction on the site as well as in our decision to 
list the words from each discipline on the main page of 
the sift results. 
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DISSEMINATION AND 
REVIEWS OF WORDSIFT 



WordSift \Nas launched to the public on January 1 5, 2009 
through a distribution list of about 500 people in the field 
of education, primarily with interests in supporting ELL 
students. Since then, it has spread mostly through word 
of mouth, links from other websites, Twitter, and formal 
reviews in publications and websites. As of February, 
2011, WordSift has had approximately 1 20,000 absolute 
unique visits, over 500,000 page views, with an average 
of longer than 3 minutes per visit. Predictably, the uses 
are higher on weekdays than on weekends. 

The website has received attention through a variety 
of channels including the following (that have driven 
considerable traffic to us): The School Library Journal, 
AccELLerate from the National Clearinghouse for 
English Language Acquisition (which has also scheduled 
a featured Webinar presentation on WordSift on October 
23, 2009), ASCD Express, and www.killerstartups.com 
that featured us on June 2, 2009, and I have had a steady 
stream of requests for permission to place screenshots 
on blogs. 

We have had excellent reviews posted on our website 
through Kampyle, a feedback mechanism that attempts to 
sample 1 0% of our users. Out of over 500 responses we 
have received, the average response is over 4.5 on a scale 
of 5. Comments left on the site are also overwhelmingly 
positive. I have also received e-mail feedback, although 
this is more likely to be positively biased. 



The main negative concern expressed by users had to 
do with potentially inappropriate pictures or videos that 
appear with respect to explicit violence or sexual content. 
We addressed this concern with a strong filter on Google 
Images, and we also removed the thumbnails from fhe 
video search results, which are less capable of being 
filtered. Since instituting those changes, we have not had 
any complaints about inappropriate materials, although 
we always encourage teachers to preview the materials 
whenever possible. 

Most importantly, teachers appear to be maintaining their 
use of the tool, and many are playing an active role in 
the process of implementation. Most of the teachers we 
have been working with have volunteered to develop 
workshops for other teachers on uses of WordSift, 
and have suggested that they reach out to teachers 
in Social Studies as well as high school and upper 
elementary teachers. These teachers have also presented 
at the California Association for Bilingual Education, 
the California Science Teachers Association, and the 
upcoming annual meeting of the National Science 
Teachers Association. 
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In addition to developing WordSift and looking at how 
it could be applied in the classroom, our effort also 
extended to the development of research to evaluate 
the effectiveness of WordSift on student reading 
comprehension of science text. In the spring of 2010, 
we implemented a systematic study with the science co- 
developers to see whether previewing a science text with 
WordSift would improve the reading comprehension of 
their students. Through collaborative effort with the SERP 
co-developer teachers, we came up with an experimental 
design that was realistic in the context of their instruction. 
We describe the experimental study below. 

THE EXPERIMENT 

This experiment was spread out over two class sessions 
of approximately 50 minutes, and each student participant 
experienced both the treatment and control conditions. 
In the treatment condition, teachers led students in 
interacting with WordSift, into which a particular passage 
from a grade-level science textbook had been entered. 
Students completed a preview worksheet based on 
the WordSift display. Then students read the passage 
individually and answered a series of comprehension 
questions about what they read. In the control condition, 
teachers also led students in completing a preview 
worksheet for a science textbook passage, this time 
without the benefit of the WordSift display. Then, as in 
the treatment condition, students read the passage and 
answered comprehension questions about what they 
read. The texts and conditions were counterbalanced 
to eliminate text and order effects. Assignment to 
conditions occurred at the class level. All aspects of 
the experimental design process, from the selection 
of the textbook excerpts to the development of the 
comprehension assessments and scoring rubrics, were 
conducted collaboratively with the nine middle school 
teachers. 



DATA SOURCES 

Our analysis used assessment data from 479 students, 
for a total of 958 observations, nested within 1 9 
classrooms of eight different teachers. Here we report 
results for the sixth grade assessment data, which 
consists of 394 observations nested within 1 97 students 
who are enrolled in eight different classes taught by four 
teachers. 

Two assessments were designed, one for each of the 
two reading passages, one on fossil fuels and one on 
solar energy. Each assessment consisted of two open- 
response questions and two multiple-choice questions. 
Questions for the assessments were drawn from the 
science textbook itself, as well as from the suggestions 
of teachers and district content specialists. Each open- 
response question was scored on a four-point rubric, 
which was developed and refined over the course 
of repeated meetings with teachers. Multiple-choice 
questions were scored as correct or incorrect. These 
assessments were scored by teachers and researchers 
after a scoring calibration process including an inter- 
rater reliability check. Scores for assessment items one 
and two (the open-ended response items) were each 
converted to separate z-scores. Scores for assessment 
items three and four (the multiple-choice items) were 
summed, and this sum was also converted to a z score. 
We then checked the reliability of the three scales by 
computing Cronbach's alpha separately for each of 
the two assessments. For the fossil fuel assessment, 
Cronbach's alpha was 0.567, and for the solar energy 
assessment Cronbach's alpha was 0.672. 
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RESULTS 



We analyzed the reading comprehension results using 
hierarchical linear modeling (HLM). HLM allows us to 
account for the nested structure of our data, since it 
makes no assumption that observations are independent. 
Since our data consists of observations (i.e. reading 
comprehension assessment scores) nested within 
students nested within class periods, we constructed a 
three-level HLM model. 

First, to determine the portion of variability associated with 
each level - observations, students, and class periods - 
we constructed an unconditional HLM model, which 
indicated that 55% of the total variance in assessment 
scores is between students and between class periods 
(more specifically, 1 7% is between students and 38% is 
between class periods). Next, we added variables at the 
observation level: one that indicates whether a particular 
observation occurred with or without the treatment 
and another that indicated whether the observation 
occurred on day 1 or day 2 of the experiment. Then we 
entered student-level variables, controlling for gender 
and participation in gifted programs, as well as whether 
the student experienced the treatment on day 1 or day 
2 of the experiment and which text the student read 
first. 3 Additionally, we checked whether these student- 
level variables mediated the treatment effect. Finally, we 
entered classroom-level variables, controlling for schools' 
standardized test scores and schools' ethnic diversity. 

The HLM results suggest that WordSift did not have a 
significant effect on students' reading comprehension (p 
>.05). 



Two student-level variables - gender and participation 
in the gifted program - were significantly related to 
students' reading comprehension scores. 4 HLM also 
allows us to enter classroom- and school-level variables, 
which indicated that schools' standardized test scores 
and schools' ethnic diversity also have a significant 
relationship with students' reading comprehension scores 
(p<001). Thus, it appears that the variation we observed 
here in reading comprehension assessment scores is not 
significantly related to differences based on exposure to 
WordSift but rather to differences among characteristics 
of the students and the classes and schools in which they 
are enrolled. 

Interestingly, in the HLM model, gender significantly 
mediated the treatment effect, resulting in a significant 
negative treatment effect for girls and a slightly positive 
(though not significant) treatment effect for boys (p>.05). 
After controlling for the other observation-, student-, 
and school-level factors in our model, boys scored an 
average of .055 points higher on reading comprehension 
assessments after using WordSift. Girls, however, scored 
an average of .586 points lower after using WordSift. 
Prior research suggests that girls tend to experience 
and interact with educational technology differently boys 
and typically have less experience with and less positive 
attitudes towards computers. For future iterations of the 
web-based vocabulary tool, we may choose to pilot test it 
specifically with girls to get their feedback about design 
and features that appeal to them. 



3 The indicator variables for text order, treatment order, and assessment day were not significant in any of our models. Nonetheless, we retained these variables in 
our models to control away any possible variability associated with the design itself. 

4 Although we are particularly interested in the effect of the web-based vocabulary tool for English Langualge Learners, this sample of sixth graders contained only 
seven ELLs (3.6% of the sample), limiting our ability to detect treatment effects specific to this population. In future experiments, we will include classrooms with 
more ELLs. 
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FIGURE 2: EFFECT OF WEB-BASED VOCABULARY TOOL 
ON READING COMPREHENSION ASSESSMENT SCORES BY GENDER. 



Treatment Effect for Girls Treatment Effect for Boys 





0 = Control, 1 = Treatment, other variables held constant at their means 



CONCLUSION AND NEXT STEPS 
IN EVALUATION PROCESS 

Although the results of this particular experiment were 
not significant except for one possibly interesting finding 
regarding gender, it is important to remember that this 
is just one implementation of WordSift (a whole-class 
demonstration, followed by a reading passage and overall 
comprehension of the text), and only a first attempt to 
investigate the tool's impact on learning. The limited 
number of English learners in the sample analyzed here 
prevented us from fully exploring how the web-based 
vocabulary tool affects the reading comprehension of 
this population, and in future experiments we hope to 
include classrooms with more English learners. 

Moreover, this particular evaluation was only designed 
to measure short-term vocabulary development, not 
retention. It may be that both the treatment and comparison 
groups did about equally well reading a passage and 
then answering questions, but future research would 



benefit from studying whether instructional technology 
such as the WordSift tool, which is designed as an 
interactive platform to engage students more fully in the 
learning process, has any differential long term impact on 
academic vocabulary retention. 

Future evaluations may also be undertaken to determine 
whether WordSift shows an effect on students’ reading 
comprehension when used in other ways. For example, 
does WordSift show an effect if students interact 
with it individually in a computer lab before reading? 
We may also investigate whether different lengths of 
interaction with WordSift produce different effects on 
reading comprehension. Each of the possible functions 
of WordSift can also be tested independently, and 
depending on the results, the usage as well as the 
website itself can be modified accordingly. 
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Over the course of our work developing WordSift, 
documenting its use in the classroom, and evaluating its 
impact on student learning, we were aiming to harness 
the power of web-based technology to transform the 
experience of classroom learning in content areas 
to include the meaningful development of academic 
vocabulary and literacy - particularly for English 
Language Learners. From the perspective of a researcher 
committed to finding innovative ways to support student 
learning, it is clear that research and investment in 
technology holds great potential as a means of improving 
academic achievement and student engagement. 

Yet it is worth noting that that the two pillars of the project 
- language and technology - played the appropriate role 
of being the background rather than the foreground of 
the discussion, as they are really means to the end of 
access to core academic content. That is to say, one 
of the greatest challenges in addressing the needs of 
students in general, and English Language Learners in 
particular, is the problem of having content teachers see 
themselves as playing a role in the language development 
of their students, in addition to the development of 
content knowledge (Dutro & Kinsella, 201 0). Likewise, it 
is easy to get enamored by the technology itself, and to 
lose sight of the fact that the technology should be in the 
service of learning goals. 

The transparency and user-friendliness of WordSift 
was designed to do just that: keep the content in the 
foreground without unnecessarily calling attention to the 
distracting details of linguistics or technology. 

The tool also seeks to account for the realities of urban 
public schools, where teachers vary considerably both 
in their comfort level in using technology and in the 
technological capacity of their classrooms and computer 
labs. An advantage of WordSift, in retrospect, was the 



fact that it only requires an Internet connection and 
an updated browser. Implementation does not require 
licensing or special hardware, and we worked hard to 
address early technology glitches and streamline the 
design to make it fast, simple to use, and engaging. 
Clearly, this is an advantage for widespread adoption, 
and we hope that WordSift is well positioned to reach 
and support even more classrooms in the coming years. 

This project was also, at its core, a study in collaboration, 
and the active role played by the teachers through each 
phase of development, implementation, and evaluation of 
the WordSift tool cannot be overstated. Disappointing 
as the results were of the first experiment in terms of 
effectiveness, the project has stimulated an interest on 
the part of the teachers and the district in continued 
exploration of the iterative cycle of instruction and 
evaluation. As one teacher enthusiastically remarked at 
one point when we were analyzing the data on the spot 
during one of our work sessions, “this is the most exciting 
form of professional development I've ever had!” 

This, in itself, is a particularly important lesson learned 
in our work in San Francisco given the vital role that 
hands-on work with teachers plays in supporting the use 
of technology in the classroom, as with any instructional 
tool or program. Because of the dynamic nature of the 
collaboration, I believe that even the teachers who were 
not initially comfortable with technology quickly came 
around and developed an enthusiasm for the tool. This 
created an environment where, as a team, we became 
genuinely interested in tracking its potential uses in the 
classroom and assessing its impact on student learning. 
Although incorporating this level of collaboration in the 
design and rollout of instructional technology entails a 
great amount of effort and commitment, overall the work 
became more meaningful to both the district and to the 
research team. 
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